Gauging the Gaps 



A Deeper Look at Student Achievement 




TO THE POINT 



Educators and policymakers need the best information available to measure achievement gaps and make progress in closing them. 

A comprehensive and accurate look at achievement gaps requires at least four perspectives: simple gap narrowing, gains across 
groups, gap size, and group comparisons. 

These perspectives on gap closing can reveal meaningful differences among states, districts, and schools. 





To gain a true picture of gaps in student achievement requires 

looking at the data from at least four different perspectives: 

• Simple Gap Narrowing: Have absolute gaps in mean 
performance between groups decreased overtime? 

• Progress for All: Have all groups of students gained over time? 

• Gap Size: What is the current size of the gap between groups? 

• Group Comparisons Across Jurisdictions: How does each 
group of students currently perform compared with their 
counterparts in other jurisdictions? 
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L eaders in schools, districts, and states, along with policymakers in 

Washington, D.C., are focusing new energy on closing long-standing 
gaps in performance that separate low-income students and students 
of color from others. It's critically important that their efforts suc- 
ceed — for students, their families, their communities, and for our democracy 
and future prosperity. Indeed, one recent estimate compared the achievement 
gap's impact on our economy to that of "a permanent national recession." 1 

Given the high stakes, educators and policymakers need the best infor- 
mation available to learn the true measure of achievement gaps and make 
progress in closing them. 

This brief explores the pitfalls in gauging gaps simplistically and suggests 
four ways to gain a more sophisticated, comprehensive, and accurate picture. 

It then illustrates these approaches using data from the National Assessment 
of Educational Progress (NAEP) to show how some states are making progress 
in closing gaps — and others are not. 

PERSPECTIVES ON GAP CLOSING: CAUTIONARY TALES 

The most common way of measuring gaps is by simple subtraction: the perfor- 
mance of white students minus the performance of African-American students 
equals the African American-white achievement gap. If the resulting number 
is decreasing over time, then the gap is closing; if that number is stagnant or 
growing, then the gap is not closing. 

It turns out, however, that without additional information this formulation 
can be misleading. 

For example, from 2003 to 2009, Georgia and West Virginia both nar- 
rowed the gap separating African-American and white students on the NAEP 
eighth-grade math test. Before celebrating, though, we would want to be sure 
that Georgia did so by making progress with African-American and white 



students alike. (It did.) West Virginia, however, narrowed the gap in a less 
desirable way: African-American students improved, but the performance of 
white students stagnated. Clearly, then, how gaps close is important. All groups 
should be making gains. 

Two states or districts that are succeeding in narrowing gaps could differ in 
another way: They could have remaining gaps of very different sizes. Delaware 
and the District of Columbia both made significant 2 progress in closing their 
Latino-white gaps on the NAEP fourth-grade reading test, but the gaps that 
remain differ markedly. Despite its progress, the District of Columbia's gap 
remains more than three times as wide as Delaware's. And size matters. With- 
out that perspective, our view of gap narrowing could miss the substantive 
differences in achievement that still separate some young people — and some 
jurisdictions — from others. 

What's more, states and districts that are narrowing gaps can have strikingly 
different performance levels for the same groups of students. Take Florida 
and Louisiana, for example. Both have significantly narrowed their African 
American-white gaps in fourth-grade math, but African-American students 
in Florida perform substantially above their counterparts in Louisiana. The 
ten-point difference between the states equates to roughly a year's worth of 
learning for students. Thus, comparing the performance of student groups 
with the performance of those same groups in other jurisdictions provides a 
more complete picture. 

In sum, each question asked in isolation provides some useful informa- 
tion. But together they provide a much better foundation for understanding 
patterns of student achievement. 



Anna Habash Rowan is a policy analyst, Daria Hall is director of K-12 policy 
development, and Kati Haycock is president ofThe EducationTrust. 
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FOUR PERSPECTIVES 

on State, District, or School Performance 

To gain a true picture of gaps in student achievement — whether on state tests, 
national tests, high school graduation rates, or almost any other measure — 
requires looking at the data from at least four different perspectives: 

• Simple Gap Narrowing: Have absolute gaps in mean performance 
between groups decreased over time? 

• Progress for All: Have all groups of students gained over time? 

• Gap Size: What is the current size of the gap between groups? 

• Group Comparisons Across Jurisdictions: How does each group of 
students currently perform compared with their counterparts in other 
jurisdictions? 

To illustrate what the data look like from each of the four perspectives, 
we'll present data on the performance of low-income and higher income stu- 
dents on the NAEP fourth-grade reading assessment. We chose NAEP because 
its results are comparable across states, thus allowing valid comparisons that 
would be unavailable using results of each state's own assessments. 3 Unlike 
state assessments, which vary greatly in quality and rigor, NAEP is widely seen 
as a high-quality assessment that reveals how well states are doing in boosting 
student achievement. 

Of course, a thorough understanding of achievement patterns in a state 
requires consideration of all subjects, grades, and student groups for which 
data are available. To enrich our picture, then, we conducted a more com- 
prehensive analysis to identify the states that have consistently high perfor- 
mance — and consistently low performance — for all groups of students on the 
NAEP tests that are available for all states over time: fourth and eighth-grade 
reading and math. The results of these more comprehensive analyses are 
displayed in the "Top and Bottom Performers" tables within each section that 
follows. In addition, we provide data on the performance of all states on each 
of the four perspectives in the tables that appear at the end of this document. 

Finally, each section includes an example that illustrates how a school or 
district can apply that perspective in analyzing its performance. 
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PERSPECTIVE 1: SIMPLE GAP NARROWING 

Have absolute gaps in mean performance between groups 

decreased overtime? 

Nationwide, low-income students and students of color perform, on average, 
below their peers. So it is imperative to evaluate whether we're helping these 
young people catch up. 

From 2003 to 2009, fourth-grade reading performance for low-income and 
higher income students alike edged up slightly nationally, though the gap 
separating these groups of students did not change. This national picture con- 
ceals varying rates of gap-closing among the states. On the high end, Illinois, 
Virginia, and Florida significantly narrowed the gap between low-income and 
higher income students in fourth-grade reading. Gaps in these states narrowed 
by at least five points. At the other end of the spectrum were Vermont and 
Oregon, where gaps actually grew significantly. 

Applying this gap-closing perspective across groups, subjects, and grades 
reveals a broader picture of state performance. Six states — Delaware, Florida, 
Georgia, Louisiana, New York, and West Virginia — and the District of 
Columbia leap to the top of the "gap narrower" chart. In other words, these 
seven have narrowed proportionately more of the gaps between groups than 
have most states. In contrast, Oregon, Pennsylvania, Utah, Vermont, and 
Washington were clear laggards, with less progress in closing gaps — and in 
fact, more gap widening — than anyplace else in the country. 

Perspective 1: Simple Gap Narrowing 

NAEP Reading and Math, Grades 4 and 8, All Groups 

Top States District of Columbia, Delaware, Florida, Georgia, 

Louisiana, New York, West Virginia 

Bottom States Oregon, Pennsylvania, Utah, Vermont, Washington 

This same approach can spotlight problems and progress within states 
or districts, too. State policymakers and advocates can use state assessment 
results to examine which districts are narrowing gaps and which are widening 
disparities. Similarly, district leaders could use state assessment results to look 
at gap-closing progress across their schools. 
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Determining the Top and Bottom States for Each Perspective 

Under each perspective and for each state, we determined the total number of ac- 
cumulated points across fourth and eighth-grade reading and math. This number 
was then divided by the points each state could possibly accumulate. Finally, 
states were ranked according to the resulting ratios. 

Here's how points were awarded: 

Simple Gap Narrowing: States gained a point for each gap that narrowed signifi- 
cantly and lost a point for each gap that widened significantly. 

Progress for All: States gained a point for each subgroup that improved signifi- 
cantly and lost a point for each subgroup that declined significantly. 

Gap Size: States gained a point for each gap that was significantly smaller than 
the national average and lost a point for each gap that was significantly larger 
than the national average. 

Group Comparisons Across Jurisdictions: States gained a point for each subgroup 
that was performing significantly above the national average and lost a point for 
each subgroup that was performing significantly below the national average. 



Leaders of the Godwin Heights school district in the down-on-its-heels 
industrial town of Wyoming, Mich., did just that and learned that one school, 
North Godwin Elementary, was doing a much better job of erasing the aca- 
demic disparities that separate some students, including English-language 
learners, from their peers. 4 Based on the school's success, district leaders 
tapped the North Godwin principal to lead gap-closing efforts districtwide. 

PERSPECTIVE 2: PROGRESS FOR ALL 
Have all groups of students gained over time? 

Our country needs to improve achievement for all students and accelerate 
gains for those who lag behind. Reading performance for low-income fourth- 
grade students nationwide inched up by four points from 2003 to 2007. This 
represents movement in the right direction but at far too slow a pace. 

Some states, however, improved much more rapidly than the nation as a 
whole. Reading scores of low-income students in Alabama increased by 11 
points — considered roughly a full year's additional learning. Scores for low- 
income students in Maryland, New Mexico, and Pennsylvania rose by eight 
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points, and in each of these states, the performance of higher income students 
improved significantly as well. 

As with the case of gap narrowing, the data for other states are troubling. 

In both Oregon and West Virginia, for example, reading scores of low-income 
students dropped by six points from 2003 to 2007, roughly equivalent to a 
half-year's learning. And in both states, the performance of higher income 
students did not increase significantly. 

Moving from examining fourth-grade reading performance to reviewing 
state performance across the NAEP-tested grade levels and subjects, a dif- 
ferent set of states emerge as leaders in making improvements for all tested 
groups. Eight states — Georgia, Massachusetts, Maryland, New Jersey, 

Nevada, Pennsylvania, Texas, and Vermont, together with the District of 
Columbia — showed significant improvement across groups. On the other 
hand, five states — Michigan, North Carolina, Oregon, South Carolina, and 
West Virginia — saw improvement for a much smaller proportion of their 
tested student groups than did their counterparts and were more likely to have 
experienced declines. 

Perspective 2: Gains Across Groups 

NAEP Reading and Math, Grades 4 and 8, All Groups 



Top States District of Columbia, Georgia, Massachusetts, Maryland, 

New Jersey, Nevada, Pennsylvania, Texas, Vermont 

Bottom States Michigan, North Carolina, Oregon, South Carolina, West Virginia 

By using state assessment results, educators and policymakers can look at 
the rate of improvement across districts and schools to find out which ones 
are boosting student performance for all groups and where performance has 
stagnated or even dipped. 

Ware Elementary School, a public school on the Fort Riley military base 
in Kansas, provides a dramatic story of improvement. In 2001, this racially 
diverse school was one of the first in Kansas to be designated as "on improve- 
ment," meaning that performance for all groups was among the lowest in the 
state. By 2008, Ware had improved so much that nearly every student was 
meeting standards. For example, in fifth-grade reading, 98 percent of white 
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students, 100 percent of African-American students, and 98 percent of low-in- 
come students met state standards. 5 Examining the steps Ware took to achieve 
such improvement could yield valuable lessons for educators and policymak- 
ers everywhere. 

PERSPECTIVE 3: GAP SIZE 

What is the current size of the gap between groups? 

In addition to examining how far a state has come in closing the gaps and 
looking at whether all students are gaining, it's important to know the extent 
of the gaps that remain. The current-year size of a state's gap suggests how 
far we have to go until race and income no longer play a significant role in 
student achievement. 

Nationally and in every state, low-income students trail their higher 
income peers in reading performance. Yet a closer scrutiny of state data shows 
that some are closer to achieving equitable results than others. In five states — 
North Dakota, Delaware, Hawaii, Oklahoma, and Wyoming — the gap in 
reading achievement between low-income and higher income fourth-graders 
is 18 points or less. That's approximately half the size of the gap in Connecti- 
cut, the state with the largest gap. 

Because gap size can vary from group to group, subject to subject, and 
grade level to grade level, it is important to weigh all three to get a compre- 
hensive estimate of the work that remains. By that measure, eight states stand 
out for smaller-than-average gaps: Delaware, Florida, Kentucky, Maine, 
Oklahoma, Vermont, West Virginia, and Wyoming. By contrast, five other 
states — California, Connecticut, Illinois, Rhode Island, Wisconsin — as well 
as the District of Columbia have gaps between groups much wider than those 
of the country as a whole. 

Perspective 3: Gap Size 

NAEP Reading and Math, Grades 4 and 8, All Groups 

Top States Delaware, Florida, Kentucky, Maine, 

Oklahoma, Vermont, West Virginia, Wyoming 

Bottom States California, Connecticut, District of Columbia, 

Illinois, Rhode Island, Wisconsin 
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In much the same way, looking at gap size across districts and schools can 
enrich any picture of student achievement. Although doing so often reveals 
findings similar to those of the state as a whole (low-income students per- 
forming below higher income students and students of color performing 
below white students), such an analysis typically turns up schools or districts 
with much smaller gaps than others and, occasionally, even places where the 
gap is "reversed." 

For example, at Roxbury Preparatory Charter School, which serves Boston's 
Roxbury and Dorchester neighborhoods, one-quarter of low-income students 
performed at the advanced level on the seventh-grade English Language Arts 
exam in 2009, compared with 14 percent of the school's higher income stu- 
dents. 6 Sometimes called a "reverse" or "negative" gap, this exceptionally high 
performance for low-income students shows what is possible for such stu- 
dents when schools really focus — as Roxbury Prep does every single day — on 
providing all students with access to a rich, engaging curriculum and the sup- 
port necessary to successfully meet high expectations. 

PERSPECTIVE 4: GROUP COMPARISONS ACROSS JURISDICTIONS 
How does each group of students currently perform 
compared with their counterparts in other jurisdictions? 

Although many assume that certain groups of children perform about the 
same no matter where they attend school, comparisons of group performance 
across jurisdictions can reveal striking differences. In fact, dramatic variations 
in the achievement of similar groups of children occur across states or from 
one district to the next. 

On the NAEP fourth-grade reading assessment, for example, low-income 
students in a diverse group of states — from sparsely settled states such as Mon- 
tana, North Dakota, Wyoming, and Maine to the more densely populated 
Delaware, Massachusetts, Florida, and Virginia — perform from ten to 20 
points ahead of their counterparts in Alaska, Nevada, Arizona, and Califor- 
nia, where performance for low-income students remains lowest. That point 
spread represents one to two years' worth of learning. 

As for group performance across grade levels and subjects, six states clearly 
lead the rest. In these states — Delaware, Massachusetts, Maryland, New Jersey, 
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Texas, and Vermont — low-income and minority students typically perform 
substantially higher than such students in other states. At the same time, five 
states are clearly lagging on this indicator: Arizona, California, Louisiana, 
Mississippi, and Nevada. In these states, low-income students and students of 
color typically perform well below their counterparts elsewhere. 

Perspective 4: Interjurisdicational Group Comparisons 

NAEP Reading and Math, Grades 4 and 8, All Groups 

Top States Delaware, Massachusetts, Maryland, New Jersey, 

Texas, Vermont 

Bottom States Arizona, California, Louisiana, Mississippi, Nevada 

At the local level, probing the results of state assessments can reveal varia- 
tions in group performance across districts and schools. Such a comparative 
look at the data reveals that some districts and schools are succeeding better 
than others in educating similar groups of students to high levels. Likewise, 
the data may show that districts or schools perform well for some student 
groups and fall short for others. 

Fairfax County, Virginia, for example, encompasses one of the state's 
wealthiest and highest performing school districts. On the 2008 Virginia 
Standards of Learning fourth-grade reading test, Fairfax students tied for sixth 
among 48 districts that either are adjacent to the county or are among the 
largest in the state. But that commendable performance for Fairfax's fourth- 
graders overall masks glaring differences in group performance. 

For example, while Fairfax's white fourth-graders performed much better 
than white students in most Virginia school districts (tied for third among the 
48 other districts), its African-American fourth-graders actually rank below 
the state average for their peers. In fact, African-American students in 20 other 
large Virginia districts — including Richmond, Virginia Beach, and Chesapeake 
City, three districts with far higher poverty rates — outperformed African-Amer- 
ican fourth-graders in Fairfax. 7 
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BRINGING IT ALL TOGETHER 

What can we learn about success in raising achievement and 
narrowing gaps by bringing these four perspectives together? 

Weaving together insights from all four perspectives can create a far more 
complete and nuanced picture of state progress in closing the gaps. In fourth- 
grade reading, for example, both Florida and Virginia stand out in terms of 
performance for low-income students. In both states, low-income students 
improved more rapidly than their higher income peers, thus narrowing the 
gap significantly. Not only have these states raised achievement and narrowed 
gaps, but they are among the country's top performers in terms of achieve- 
ment of low-income students and in regard to smallest gap size between 
low-income and higher income students. It is important to note, however, 
that neither state significantly increased the performance of higher income 
students from 2003 to 2007. 

Another group of states — Idaho, Kansas, Montana, New Hampshire, 
North Dakota, and Oklahoma — have not significantly narrowed the fourth- 
grade reading gap but are showing other good progress for low-income 
students. Low-income students in these states improved significantly, and their 
2007 performance ranks among the highest in the nation. What's more, these 
states can point to gaps between low-income students and higher income stu- 
dents that are among the smallest in the nation. In Idaho, Kansas, and North 
Dakota, the performance of higher income students increased significantly, 
while in Montana, New Hampshire, and Oklahoma, they did not. 

The picture is more worrisome in other states. In Alaska, Arizona, 
and South Carolina, performance of low-income fourth-grade students is 
among the worst in the country in reading, and there has been no significant 
improvement since 2003. Low-income students in Oregon, sadly, rank among 
the lowest performing in the country and actually have lost ground since 
2003. Meanwhile, the gap separating these students from their higher income 
peers has grown significantly. 

To be sure, the four-perspective approach to achievement and gaps creates a 
more complex picture than does looking at any one indicator in isolation. But 
failing to apply all four analyses may cause education leaders and policymak- 
ers to interpret data in ways that are incomplete, if not misleading. 
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It's unlikely that any state, district, or school will rise to the top in every 
one of these analyses, especially if additional grades, subjects, and student 
groups enter the mix. Yet the multiple perspectives unquestionably show that 
some places have much better track records than others concerning equity and 
achievement for all. 

Looking across subjects, grades, and groups, Delaware, Florida, Massa- 
chusetts, Texas, and Vermont emerge as clear leaders. These states differ in 
many ways, including size and diversity. Looked at from our four perspectives, 
though — absolute gap narrowing, gains across groups, remaining gap size, 
and group performance compared with other jurisdictions — these states have 
compiled the best track records to date. 

On the other end of the spectrum appear the laggards on our composite 
measure — Arizona, California, Michigan, Mississippi, and Rhode Island. 
These states, regrettably, stand apart from others in compiling the worst track 
records in closing the achievement gaps. 

Bringing It All Together: Raising Achievement and Narrowing Gaps Across Groups 8 

NAEP Reading and Math, Grades 4 and 8 

Top States Delaware, Florida, Massachusetts, Texas, Vermont 

Bottom States Arizona, California, Michigan, Mississippi, Rhode Island 

BEYOND NAEP 

Although we have used NAEP data to illustrate the four perspectives on 
performance and gap closing, a richer picture of achievement certainly would 
emerge by looking at other data in those ways. State assessment data can pro- 
vide information about performance and gaps in grades three through eight 
as well as in high school, whereas state-level NAEP information is limited 
only to grades four and eight. Using state assessment data also would allow an 
exploration of results in other subjects, such as science. 

Likewise, educators and policymakers could find it useful to look not only 
at score averages but also at the percentage of students meeting or exceeding 
state standards — and to do so over a longer time, where comparable data are 
available. 
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A cautionary note: An honest look at high school assessment data requires 
the extra step of scrutinizing the graduation rates of each subgroup. Other- 
wise, decision makers run the risk of seeing narrowing gaps when, in reality, 
lower performing students are dropping out of school before taking the high 
school assessment. 

On the positive side, this more sophisticated analysis can yield examples 
of success in achievement and high school completion alike. For example, 
among African-American secondary school students in New York State in 
2008, just 59 percent met state math standards, and just 55 percent had 
graduated. But Elmont Memorial Junior-Senior High School, a large, com- 
prehensive high school serving mostly African-American and Latino students 
just outside of New York City, generated much higher numbers. Ninety-three 
percent of Elmont's African-American students met state math standards, and 
97 percent graduated by 2008. 9 Schools like Elmont show what's possible and 
provide roadmaps for how to get there. 

CONCLUSION 

As the country grapples with the challenge of closing gaps in achievement, 
multiple perspectives on performance data can inform the discussion. Track- 
ing the narrowing of gaps between groups, gains for all groups, gap size, and 
comparisons between jurisdictions can provide education leaders with a true 
picture of the inequity they seek to reduce and, ultimately, end. 

As the data show, meaningful differences between states — as well as 
between districts and schools — emerge when looking at performance in these 
ways. These differences should be considered when evaluating how much state 
and local leaders have advanced academic equity to date, as well as their readi- 
ness to make additional progress. 

A frank assessment of the effectiveness of our efforts and a commitment 
to measuring accurately the effects of any new policies and programs will go 
a long way in closing achievement gaps. By considering multiple perspectives 
on available data, educators and policymakers can better judge whether efforts 
to lift today's students will produce the better educated citizenry our country needs. 
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NOTES 

1 McKinsey & Company (2009). "The Economic Impact of the Achievement Gap in 
America's Schools. " www.mckinsey.com/clientservice/Social_Sector/our_practices/Educa- 
tion/Knowledge_Highlights/Economic_impact.aspx 

2 Throughout this brief, the term "significant" refers to statistical significance, as deter- 
mined by the National Center for Education Statistics NAEP Data Explorer. 

3 Each state currently has its own set of reading and math assessments for grades three 
through eight and high school. These assessments are based on state-developed sets of 
standards, which are unique to that state. Thus, what it means for a student to meet one 
state's standards is not the same as what it means to meet another state's standards. 

4 See Michigan Department of Education, https://oeaa.state.mi.us/ayp/ 

5 See Kansas Department of Education, http://online.ksde.org/rcard/ 

6 See Massachusetts Department of Education, http://profiles.doe.mass.edu. 

7 See Virginia Department of Education, https://plpe.doe.virginia.gov/datareports/assess_ 
test_result.do. 

8 Determining top and bottom states for bringing it all together: For each state, the total 
number of points earned across all four indicators was divided by the total points pos- 
sible. States were then ranked according to the resulting ratios. 

9 See New York State Testing and Reporting Accountability Tool, https://www.nystart.gov/. 
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NAEP Grade 4 Reading (Key: AA = African American, WH = White, NA = Native American, LA = Latino, LI = Low Income, HI = High Income) 



Yes, statistically significant 



No, statistically significant 



Yellow No change/At national average White Data unavailable 



Did achievement increase from 2003 to 2007? I Were 2007 gaps smaller than the national average? I Was 2007 achievement above the national average? 
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NAEP Grade 4 Math (Key: AA = African American, WH = White, NA = Native American, LA = Latino, LI = Low Income, HI = High Income) 



^ Yes, statistically significant No, statistically significant 

| Yellow | No change/At national average | White | Data unavailable 



Did the gap narrow from 2003 to 2009? 



Did achievement increase from 2003 to 2009? | Were 2009 gaps smaller than the national average? Was 2009 achievement above the national average? 



I NA -WH 


I LI-HI | 


1 AA 


LA 


NA 


I WH 


LI 


HI 


+3 


0 


+6 


+6 


+2 


+5 


+6 


+6 




0 


+3 






+5 


+4 


+4 


+9 


0 


+4 


+4 


-2 


+7 


+6 


+6 


1 -1 


+1 


+7 


+3 


+5 


1 +3 


+2 


+2 




THE EDUCATION TRUST I GAUGING THE GAPS I JANUARY 2010 



9 






















































































NAEP Grade 8 Reading (Key: AA = African American, WH = White, NA = Native American, LA = Latino, LI = Low Income, HI = High Income) 



Yes, statistically significant No, statistically significant 

| Yellow | No change/At national average | White | Data unavailable 



Did the gap narrow from 2003 to 2007? I Did achievement increase from 2003 to 2007? I Were 2007 gaps smaller than the national average? 
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NAEP Grade 8 Math (Key: AA = African American, WH = White, NA = Native American, LA = Latino, LI = Low Income, HI = High Income) 



J Yes, statistically significant No, statistically significant 

| Yellow | No change/At national average | White | Data unavailable 
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The Education Trust promotes high academic achievement for all students 
at all levels— pre-kindergarten through college. We work alongside parents, 
educators, and community and business leaders across the country in 
transforming schools and colleges into institutions that serve all students 
well. Lessons learned in these efforts, together with unflinching data 
analyses, shape our state and national policy agendas. Our goal is to close 
the gaps in opportunity and achievement that consign far too many young 
people— especially those who are black. Latino, American Indian, or from 
low-income families— to lives on the margins of the American mainstream. 
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